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ABSTRACT 



As institutional research or outcome assessment 



becomes more commonplace in institutions of higher education, the 
issue of what opportunities this presents to researchers desiring to 
move past relatively routine reporting into the more creative aspects 
of the profession becomes more prominent. Several possible roles 
emerge as a result of the assessment implementation process: 
Implementation Team Coordinator; Institutional Planner; Departmental 
Activity Facilitator; and Assessment Data Gatherer. Eacn role varies 
in its attractiveness in assisting professional development amd what 
is usually expected from individuals within these roles. Of 
particular interest are the two most common roles the institutional 
researcher will be as)ced to play in terms of institutional 
effectiveness and outcomes assessment: the Assessment Data-Gatherer 
Role and the Institutional Plani.er Role. Also included are two 
assessment matrixes: one is an example of an assessment plan for an 
accou:itin9 department and the other is a matrix showing whose 
responsibility it is to provide data, what type of data, and for 
which departments or services. Contains 19 references. (GLR) 
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The second, now common, context within which 
implementation of institutional effectiveness and out- 
comes assessment takes place is a state mandate for 

such activity. The impetus for such actions on the part 
of governors, le^^islatures. and state governing t>oards 
can t>e traced to dellt>enitions which took place In the 
mid-1980s and which ultimately were reflected in the 
National Govornors' Association report. Time for 
Aes(i/fs— t99t Report on Education (1986). The injunc- 
tions contained in that document have been followed in 
many states, and a soon-to-be-published report from 
the State Higher Education Executive Officers (SHEEO) 
and the EducatkKi Commission of the States (ECS) 
Indicates that 84% of state legislatures and/or governing 
boards have taken one action or another regarding 
Implementation of outcomes assessment and institu- 
tional effectiveness (States. 1990. p. 2). State mandates 
take a variety of forms, ranging from the relatively 
general Injunctions contained In the Virginia Assessment 
Plan (Virginia General Assembly. 1987) to the far more 
prescriptive set of requirement3 based upon recent 
legislation in South Carolina (South Carolina Commis- 
sion on Higher Education. 1989). Depending upon their 
nature, state mandates may have a substantial impact 
upon the role of the institutional research component 
within implementation on the campus. 

Among the more common contexts for assessment 
operations Is a campus regional accrediting effort. While 
the Southern Association of Colleges and Schools 
(SACS) Is frequently identified as having the most 
rigorous requirements regarding implementation of 
institutional effectiveness or outcomes assessment 
(Southern Association of Colleges and Schools. 1987). 
All regional accrediting associations, since 1986, have 
been required by the Council on Postsecondary Accredi- 
tation (COSA) to: 

1. Sharpen statements of mission and objectives to 
identify intended educational outcomes 

2. Develop additional effective means of assessing 
learning outcomes and results 

3. Use the self-evaluation and peer-review processes of 
accreditation as an integral part of ongoing planning 
and Institutional or programmatic change. 

Further, these mandates have been underscored by the 
Federal Government's involvement and the directive that 
accredited institutions emphasize "educational effective- 
ness" (U.S. Department of Education. 1988). 

Each of these contexts for implementation (pure 
student outcomes, state mtndpje, accreditation/insti- 
tutional effectiveness) has a considerable impact upon 
the role of the institutional research component as well 
as the overall institutional reaction. It should also be 
acknowledged that some Institutions were doing out- 
come assessments even before it t>ecame a national 
movement. Those institutions did so because they felt it 
was the right thing to do. In fact, some people believe 
that assessment efforts will have significant impact on 
program development and be of lasting value only If 
driven by internal concerns. 

Rolee wWiln tenplemeiilallon of 
mamunofMH ciiecinfen ess 

Institutions implementing institutional effectiveness 
and student outcomes aasessment frequently find that 
several roles emerge as a result of the implementation 
process: Implementation Team Coordinator. Institutional 



Planner. Departmental Activity Facilitator, and Assess- 
ment Data Gatherer (Figure 1). 



Chief Executive Officer 




Figure 1. The institutional effectiveness implementation 
team. 

Probably the greatest opportunity for professional 
growth by Institutional researchers aspiring to move 
Into more general institutional administration is to 
assume tho role of overall iinplementation team coor- 
dinator. Because implementation Is one of those activ- 
itfes (unlike registration, payroll, course scheduling, 
etc.) which can be delayed, campuses find it necessary 
to appoint an individual (as opposed to a committee) 
responsible for seeing that implementation takes place 
in a timely fashion. It is not unusual for an Institutional 
research officer, if widely respected on the campus, to 
be identified to fulfill such duties. 

The role of institutional planner Is also one which is 
frequently fil!ed by an individual heading the unit which 
may be titled "institutional research and planning" on 
the campus. It is important to understand that the type 
of planning required in institutional effectiveness or 
outcomes assessment Is different than financially 
oriented budget planning. 

The third role, departmental activity facilitator, is the 
least likely to be filled by an institutional research 
professional. This role necessitates a direct working 
relatiottship with departmental faculty in order to explain 
the requirements of institutional effectiveness or out- 
comes assessment and facilitate faculty efforts. Usually 
this role is filled by an academic administrator or a 
member of the faculty. 

At the very least, the institutional research compo- 
nents on campuses implementing institutional effective- 
ness and outcomes assessment can expect to receive 
the role of assessment data gatherer. This role involves 
the design of the assessment process and frequently the 
logistical support for the means of assessment imple- 
mented. Because this role requires the greatest level of 
technical expertise and the most physical work, and is 
most likely to be the "lightening rod" drawing tho 
greatest amount of controversy or criticism, it Is a role 
which is usually "ours to keep." 



In addition to identifying the context within which 
Institutional effectlvenoM and outcomes asaeMment are 
to be Implemented on the campus, the institutional 
researcher must ascerUIn which of the relatively distinct 
roles descrik>ed he or she is t)elng asked to fill. On some 
campuses, there will be a more or less open or "go for 
it" situation with regard to the overall coordination of 
the implementation team. On other campuses, indtvi- 
duais other than the institutional research officer will be 
identified to play the roles of implementation team 
coordinator or institutional planner. In general (depend- 
ir^^ upon the professional development goals of the 
institutional research officer), there is little to be lost in 
seeking as ^reat a role in implementation as possible. 
The nature of the role granted to the institutional 
research officer by the campus will ultimately be 
determined by the existing perception of that individual 
on the campus, the interest of others in assuming these 
roles, and any external restraints Imposed on the 
institution. 

In addition to making the decision regarding what 
role to assume, or what role the institutional researcher 
is expected to take, there are different implications that 
can arise based upon whether the assessment activities 
are handled through the expansion of the current focus 
of the institutional research component or a new institu- 
tional research component is established strictly for the 
purpose of institutional effectiveness. There are advan- 
tages and disadvantages, including cost and staffing 
considerations, in both arrangements. For a compre- 
hensive look at this topic, consult the referenced Nichols 
and Wolff New Directions chapter (1990). 

The institutional research component's involvement in 
the implementation of institutional effectiveness or out- 
comes assessment has certain organizational implica- 
tions, based primarily on its organizational reporting 
relationship. If the implementation involves a broad 
program of planning and assessment activities in all 
academic and admi'iistrative departments, then there is 
merit in having the nstitutional research component in 
the office of the chi^f executive officer of the institution. 
If the institution as a whole, and thus the institutional 
research component, focuses only on developing a 
student outcomes orientation, then it is advisable that a 
direct organizational relationship exist between the 
institutional research component and the chief academic 
officer of the institution. With respect to the institutional 
research component having an amenable working rela- 
tionship with the faculty, Ewell and Lisensky (1988) 
point out that such a close relationship is most likely to 
exist when the component reports directly to the chief 
academic officer. 

The two most common roles that the Institutional 
research officer will be asked to play in terms of 
institutional effectiveness and outcomes assessment are 
that of acsessment data gatherer and institutional 
planner. Following are detailed discussions of the two 
roles. 

The Aaseeement Data-Qatherer Role 

Understanding the context within which the institu- 
tional research officer will be asked to implement the 
assessment data-gatherer role is as important as under- 
standing the overall context within which institutional 
effectiveness Is Implemented. With what external con- 
straints Is the institutional research officer faced? In the 
case of relatively pure student outcomes assessment, 



no external constraints are imposed. In the case of 
responding to an external mandate from a state legis- 
lature or governing board, specific means of assessment 
and, potentially, even instruments will have been pre- 
scribed, and these will form the basis for the overall 
assessment effort, in the case of assessment of insti- 
tutional effectiveness, relatively little specification as to 
means of assessment Is Identified. However, the assess- 
ment process will need to be structured around the 
statements of expected departmental/program results 
established in conjunction with the institution's state- 
ment of purpose. 

Implementation of the assessment data-gatherer role 
is contingent upon a clear understanding between the 
institutional research officer and his or her supervisor 
regarding their joint expectations concerning the assess- 
ment effort. It Is necessary that this clear understanding 
be arrived at concerning the relative importance of 
assessment activities in light of overall requirements of 
the institutional research component to protect the 
institutional researcher from various competing de- 
mands for services. 

Once the context within which data relative to assess- 
ment will be gathered is Identified, then the institutional 
researcher can turn his or her attention to what means 
of assessment will be utilized. These include cognitive, 
attitudinai, and behavioral means of assessment which 
are further Identified and explained in a number of other 
publications devoted to this purpose. These include 
those by Pike (1988) and Harris (1985). In this regard, 
however, the major internal task for the institutional 
researcher charged as the assessment data gatherer is 
to identify existing assessment processes and those 
which might potentially be utilized at the Institution. 

One of the relatively few certainties regarding assess- 
ment is that all Institutions implementing institutional 
effectiveness or student outcomes assessment have 
been surprised to learn the level of assessment activities 
already taking place on their campuses in a somewhat 
unsystematic, but nonetheless constructive, fashion. 
Conducting an inventory of current assessment activities 
on the campus will identify these existing means of 
assessment while, at the same time, it will illustrate to 
the campus the amount which faculty have been doing 
on their own for some period of time. 

In addition to this inventory of existing means of 
assessment on the campus, it will be necessary for the 
institutional researcher to conduct an inventory of 
standardized means of assessment available from var- 
ious vendors. This inventory describes what means 
could be made available on the campus in a short 
period of time. In the cas« of both types of standardized 
means of assessment (attitudinai or cognitive), the 
institutional researcher must arrive at a conclusion 
relative to whether it is best to adopt such a standard- 
ized measure or to construct locally an institution- 
specific instrument, a trend which is increasing, accord- 
ing to the Et-Khawas study (1989). 

There are a number of pros and cons to consider 
when deciding whether to use standardized or localized 
mea'<s of assessment. Here, standardized refers to 
instruments made available to the institution by off- 
campus vendors, and localized applies to those instru- 
ments which are designed by faculty or sttff affiliated 
with the institution. Peter Ewell, in a 1987 New Directions 
chapter, identified a general set of pros and cons with 



4 



regard to the usa of standardized and localized assess- 
ment measurements: 

Pros or SlMidirdlif ed InalniiMnft 

1. Relatively easy to administer 

2. Acceptable In terms of faculty and staff time invested 

3. Generally less open to charges of subjectivity and 
bias 

4. Nationally normed for comparisons across institu- 
tions 

5. Validity and reliability established 

6. Processing and reporting services available. 

Cofit of Ste/Kiefdiied /nafnimenlt 

1. May not reflect the content of a specific institution's 
curriculum 

2. Scores may not be meaningful for institutional 
assessment 

3. Normative comparison scores may be inappropriate 
for curriculum evaluation 

4. May be expensive to purchase and score. 

Pros of Loco//MCf IfMmmmttt 

1. Instrument content reflects curriculum 

2. Suitable for additional analysis of results 

3. Amenable to a variety of formats (problem solving, 
essay, etc.) 

4. Considered "legitimate" by faculty (ownership). 

Conn of LocaXied ItwtmmontB 

1. Less external credibility 

2. Validity and reliability data lacking 

3. Lack of normative data for comparisons 

4. Can be costly to produce 

5. Scoring of writing samples time consuming and 
difficult. 

Once the aforementioned pros and cons have been 
considered and the decision of whether to use standard- 
ized or localized Instruments made, the next step in the 
development of the data-gathering effort is to establish 
an "Intended Outcomes/Expected Results versus Means 
of Assessment Matrix*' (Figure 2) to coordinate the 
assessment process. The design of such a matrix will 
insure that all intended outcomes/expected results are 
examined by at least one means of assess.nent. It is 
important to understand that the field of assessment Is 
at this time, and for the foreseeable future, more of an 
art than a science. That is to say, the means of assess- 
ment at our disposal are considerably less than perfect 
and, in order to compensate for their relative inade- 
quacies, it is important that multiple means be utilized 
to access each expected outcome or result. 

Once the matrix Is completed, the responsibility for 
each means of assessment will need to be established 
and documented in a chart or matrix similar to that 
shown in Figure 3. 

Thes(» actions will lead naturally to questions regard 
ing centralization versus decentralization of assessment 
support and information. This issue, though seemingly 
straightforward, has a number of perspectives relating 
to campus politics. While a persuasive case can be 
made for centralized logistical support for assessment 
from a component such as institutional research, the 
case for a centralized clearinghouse for assessment 



results is less readily accepted. The basic position 
regarding centralization versus decentralization can be 
restated by noting that decentralization of assessment 
procedures Is politically more attractive, in that results 
are not available to the central administration. However, 
it is considerably more costly in terms of both out-of- 
pocket cost and effort on the part of the faculty-Hind is 
simply less likely to take place. On the other hand, 
centralization of logistk:al support and the establishment 
of a clearinghouse for asseeament results can easily be 
justified from the standpoint of cost, reduction of the 
burden on the faculty, and easy access for accreditation 
purposes. However, ci^re must be taken that the avail- 
ability of data logistically supported from a central 
source, such as the Institutional research component, 
does not lead to abuses of the decision-making power 
from the top down within an institution. 

Ultimately, once the assessment plan is assembled 
and the instrumentation pr»p«red. assessment activHiee 
will be initiated at the institution. One can expect these 
activities to be relatively uneven in their firet iteration, 
and it will be unlikely that adequate coverage of all 
expected results by means of assessment will transpire 
for the first several years. It is more important for the 
institution to begin the trial implementation of its assess- 
ment procedures and refine them over a period of time 
than for the institution to attempt the "perfect" initial 
implementation. 

The Instttuttonal Planner Role 

The next most likely role to be served by an Insti- 
tutional researcher is that of institutional planner. 
However, as tl.e ro!e of asses&Tient data gatherer calls 
for the technical and logistical skills of the institutional 
researcher, the role of institutional planner requires a 
degree of organizational capability and Interpersonal 
relations skills beyond that sometimes found in practi- 
tioners of our profession. 

In implementation of the institutional planner role, it 
is essential to realize from the outset that the planning 
required for assessing Institutional effectiveness is 
results or outcomes related, as opposed to means or 
resource oriented. Many campuses have relatively 
sophisticated and quite capable planning mechanisms 
which focus upon the processes of acquiring and 
allocating resources to support Institutional operations 
such as teaching, research, etc.; but the institutional 
planner role focuses upon the results of the institution's 
efforts rather than these processes. As an example of 
the difference between these two perapectives on 
planning, the process- or means-oriented planner would 
be concerned with the addition of a faculty member to 
the English Department, whereas the outcomes- or 
results-oriented planner would focus upon whether 
students completing a degree at the institution could 
write a coherent paragraph. 

In addition to this recognition of the different focus of 
assessing institutional effectiveness, there are at least 
four key issues or challengea facing the individual in the 
institutional planner role: 

1. Establishment of a usable statement of purpose for 
the institution. Most institutional statements of pur- 
pose, mission, philosophy, etc. have been somewhat 
similar to "Mother. Home, and Apple Pie." This 
circumstance has existed for a number of reasons, 
including a genuine lack of focus on the part of many 
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Figure 2. Intended outcomes/expected results vs means of assessnnent matrix. 
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Figure 3. Identification of responsibilities for means of assessment. 
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institutions and a desire not to offend any consti- 
tuency by attempting to be ' all things to all people." 
Implementation of institutional effectiveness or out- 
comes assessment requires that t ase statements of 
purpose be action-oriented guid' . j the functioning 
of the institution The reason for this is that •'tsti- 
tutions must t)e able to verify the accomplisiiment of 
their goals. 

2. Establishment of departmental program statements 
of expected results or intentions. The institutional 
planner will be responsible ror seeing that the 
necessary statements are established within the insti- 
vjtion (probably in departments and programs) to 
support accomplishment of the institutional statement 
of purpose. In this capacity, the institutional planner 
will necessarily need to "map back" departmental/ 
program statements of expected results to ensure 
that ail elements of the institutional level of statemem 
of purpose are supported by one or more units within 
the institution. 

3. Ensuring the use of assessment results Probably the 
most difficult aspect of implementation of institutional 
enectiveness relates to documentation of the use of 
the assessment results. Seeing that such results are 
utilized will become one of the primary tasks of the 
institutional planner. 

4. Establishing the campus relationship of ends-oriented 
(expected results) to means-oriented (resources) 
planning. While ends- and means-oriented planning 
are obviously related, the institutional planner will 
ultimately need to address the specific nature of that 
relationship on his/her campus. Means- or resource- 
oriented planning needs the credibility gained 
through assessment of outcomes or results while 
ends-oriented planning needs a relationship to re- 
source allocation to ensure that desirable results are 
supported. 

While these challenges are substantially less technical 
in nature than those of the assessment data-gatherer 
role, they are nonetheless substantial and constitute 
nearly a full-time position on some campuses 

Conclusion regarding the Role of Institutional Research 

As indicated above, implementation of institutional 
effectiveness or outcomes assessment can be, for the 
institutional researcher, the proverbial "insurmountable 
opportunity." It is important for each of us to grasp that 
within the challenge of this implementation are roles 
which indeed represent opportunities for professional 
development. The choice for most institutional re- 
searchers whose campuses are implementing institu- 
tional effectiveness cr outcomes assessment is which 
role to seek or accept. 

William Faulkner (1950) wrote, "I decline to accept the 
end of man. ... I believe that man will not merely 
endure: he will prevail." To paraphrase Faulkner, insti- 
tutional effectiveness and outcomes assessment provide 
the institutional research officer not merely with an 
opportunity to endure within his or her institution but, in 
some cases, to prevail. 
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